Business applications under maintenance are in a constant state of change to keep pace with the changing business requirements. This results in frequent releases of the software system. An important task in the maintenance process is therefore to test the new or modified functionality and to ensure that the rest of the system unaffected by the change does not break down. Functional tests to test the changes and regression tests are executed multiple times in each test cycle. Test teams are therefore under constant pressure to reduce test cycle time, and with low test automation levels to make optimal use of available resources. Adding more resources to parallelize testing so as to reduce time, does not always help as dependencies may exist between test cases that mandate an order for test execution. The number of test cases and the test cases themselves may vary in each test cycle making accurate time and effort estimation for test cycle a challenge.
Within each release cycle, the time taken for testing and resources are estimated from the available time and efforts, using empirical data and experience. This is computed statically for each release cycle. Very little effort has been directed towards addressing dynamic time and resource estimation and arriving at effective workload distribution during test cycle.
The parallel execution of test cases results in out of order operations on the system. In such a case, if test cases use persistent data source, for example, a database across the test cases, then different execution sequences might result in different output due to the dependencies across the test cases. The impact of execution sequences of the test cases has been identified in the past where the test cases have been partitioned based on state preservation by resetting the states. The work done in the prior art creates a conflict database for out of order executions and analyzes the same to reorder test sequences. The approach assumes automation and its applicability and effectiveness for manual testing has to be studied. The experts, in the past, have used static analysis to analyze the dependencies across the JDBC queries in a program. The approach assumes that testers have full access to the systems and the source code which is often not the case in most of the projects during functional and regression test cycles.
Parallel and distributed executions of test cases are explored and used in various tools and frameworks. In these frameworks testers specify the test cases that can run in parallel. Previously, authors have proposed a framework to execute the regression test cases in a distributed manner and also some of the previous works has also shown execution of test cases in parallel and in a distributed manner on the cloud framework.
The distributions of test cases may result from different reasons such as functionality, domain understanding etc. One of the works of the prior art partitions the test cases which capture related set of bugs by analyzing the source code. However, in this case the test cases are independent and they consider the source code availability. Some have even considered resource constraints as the parameter of test suite distribution. But no one has ever considered dependencies across test cases as the constraint to be handled during workload distribution.
Further, various approaches have been discussed to achieve balanced partitions. Also, time aware test case execution schemes have also been discussed where prioritization of test cases for a given constraint is taken care of. None of the arts have identified the resource requirement for the specified time constraint.
In the view of above technical challenges an automated approach to partition the test suite for parallelization given the dependencies existing between the test cases is what is needed as a technical solution.